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1 .  OBJECTIVES 


The  purpose  of  the  quality  control  package  for  DDA  and  ODDA  was  to 
assess  the  performance  of  the  Digisonde  256  system  and  communicate  any 
errors  detected  to  the  operator.  The  quality  control  system  should  address 
the  following  points. 

(1)  The  package  should  contain  a  system/program  update  to 
inform  the  operator  of  its  current  status. 

(2)  Indicate  the  type  of  data  being  gathered  for  future  processing 
at  a  glance. 

(3)  Test  or  monitor  the  hardware,  with  special  emphasis  on  the 
testing  of  the  antenna  amplitude  and  phase  characteristics. 

(4)  For  the  onsite  system  (ODDA),  any  errors  detected  should  be 
communicated  back  to  the  remote  terminal,  and/or  flags  should 
be  set  with  the  data  gathered  to  identify  these  data  as  good  or 
bad.  This  mode  has  yet  to  be  satisfactorily  implemented,  and 
will  not  be  discussed  in  this  report. 

DDA  contains  two  graphics  windows.  These  windows  contain: 

Window  1  =  ODDA  QUALITY  CONTROL 

Window  3  =  DRIFT  VELOCITIES 

DDA  and  ODDA  display  results  in  two  text  windows  for  the  onsite  systems 
and  for  any  computers  unable  to  support  the  VGA  graphics  options.  These 
windows  display  the  quality  control  and  drift  velocities  in  ASCII  format  to 
screen.  In  the  following  sections,  the  graphical  and  text  displays  will  be 
described  in  detail. 


2 .  DDA  QUALITY  CONTROL  GRAPHICS 


An  example  of  the  DDA  quality  control  graphics  screen  is  displayed  in 
Figure  1.  The  display  consists  of  two  main  sections,  the  Antenna  status 
and  the  System  status. 
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D.D.A. 

Quality  Control 


Antenna  status 
Phase 


90 


0 


-90 


System  status 

Option  Millstone  Hill  cdts  COMPASS. 

Tape  /  File  :  Tape  online 
Data  block  :  Gathering  data 
Antenna  Err :  Mo  errors  detected 
Antenna  No  : 

Current  tine:  1990  049  00:56:42 


Amplitude 


Number  of  sources  detected 


Ave. 


Indv. 


Number  of  subcases  processed 


Figure  1.  Digisonde  Drift  Analysis  (DDA)  Graphics  Screen 
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2.1  System  Status 


The  system  status  is  further  divided  into  a  graph  and  text  messages.  The 
graph  indicates  the  total  number  of  sources  detected  for  each  subcase  (i.e. 
all  spectra  in  one  frequency  range  bin).  The  red  blocks  labeled  as  INDV  on 
the  screen,  indicate  the  total  number  of  sources  for  each  subcase.  Up  to 

100  height  spectra  are  displayed  at  any  one  time.  After  100  subcases 

have  been  processed  the  overall  number  of  sources  are  averaged  with  the 
previous  100  subcase  average  and  this  is  then  displayed  by  the  green 
block  preceding  the  first  subcase,  and  listed  as  AVE  on  the  screen.  Hence, 

the  AVE  block  is  an  accumulative  average  of  the  number  of  sources 

detected  and  gives  an  indication  of  the  characteristics  of  previous 
measurements. 

The  time  for  the  current  drift  data  block  being  processed  is  given  above 
this  graph.  This  is  useful  when  processing  data  from  tape  or  disk. 

Immediately  under  system  status,  the  OPTION  line  is  displayed.  All 
optional  messages  tell  the  operator  what  DDA  is  doing,  or  what  is  to  be 
done  in  case  an  error  is  detected.  Below  OPTION  are  the  main  system 
update  messages.  The  state  of  DDA's  program  and/or  if  any  errors  are 
detected  in  the  system  are  updated  in  these  lines. 

2.2  Antenna-  Slates 

The  two  graphs  below  the  antenna  status,  display  the  performance  test  for 
each  receiver  antenna,  cable  and  antenna  switch  combination.  These 
separate  tests  can  be  done  to  verify  if  the  antenna  system  is  working  well. 
The  first  graph  labeled  phase  may  display  the  "phase  failure  rates"  or  the 
"phase  error"  on  each  antenna.  The  test  displayed  in  this  graph  depends 
on  the  setting  of  option  *074  in  the  ODDAMENU.ONL  file. 

for  *074  option  =  1  the  phase  failure  rates  are  displayed 

for  *074  option  >1  the  phase  error  on  each  antenna  are  displayed 

The  phase  failure  rates  calibrated  by  the  0  to  100%  scale  on  the  left  hand 
side  of  this  graph  test  for  randomness  in  the  phase.  Each  time  the  phase 
characteristics  of  an  antenna  do  not  meet  certain  criteria  a  failure  is 
detected  and  recorded  for  that  record  group  and  displayed  by  a  red 
histogram  (INDV  marker).  If  an  antenna  fails  consistently  over  60  groups, 
then  the  red  bar  would  be  at  60  on  the  scale.  After  100  groups  the  INDV 
values  are  averaged  with  the  previous  100  group  average  for  a  cumulative 
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average  value  AVE  which  is  given  by  the  green  histograms  for  each 
antenna. 

Ideally,  the  phase  AVE  failure  rates  (or  %  failure  counts)  of  all  antennas 
should  be  consistently  the  same.  Any  substantial  deviation  of  failure  rate 
from  antenna  to  antenna  may  indicate  a  problem.  The  %  failure  counts 
should  never  exceed  20%  if  there  is  no  problem  with  the  system.  Similarly 
the  second  graph,  labeled  Amplitude,  also  shows  the  INDV  and  AVE  failure 
rates  for  each  antenna  but  the  amplitudes  are  tested  in  this  case.  In  the 
second  graph,  the  eighth  column  is  also  used  and  is  a  measure  of  the 
independence  of  noise  and  the  amplitudes  of  the  spectrum.  Ideally  the 
failure  rates  in  amplitude  should  always  be  low,  preferably  below  or 
around  10%,  but  definitely  never  greater  than  20%. 

Displaying  the  phase  error  on  each  antenna  by  setting  option  074  >  1  in  the 
menu  file,  the  results  are  defined  by  the  scale  on  the  right  hand  side  of 
phase  graph,  centered  on  0.  The  phase  error  test  attempts  to  estimate  the 
amount  of  phase  error  consistently  observed  on  each  antenna.  If  the 
system  is  working  well,  all  AVE  (green  histograms)  for  active  antennas 
should  be  at  0  degrees.  If  the  AVE  histogram  is  smaller  than  the  0  degree 
level,  this  indicates  a  negative  phase  error  on  the  antenna,  while  larger 
green  bars  above  0  specifies  positive  phase  errors.  This  test  works  best 
for  the  7  antenna  system  rather  than  the  4  antenna  system. 

Hidden:  A  Quality  Control  file  is  generated  every  time  DDA  or  ODDA 

updates  the  AVE  statistics,  so  that  on  start  up  DDA  or  ODDA  can 
recall  the  current  state  of  the  system.  The  file  is  named: 

QUCNTRL.DAT 

When  beginning  a  completely  new  drift  data  set,  it  would  be  advisable  to 
delete  this  file  and  begin  the  statistics  from  the  start. 


3 .  DETERMINATION  OF  %  FAILURE  COUNTS 


3.1  Amplitude.,  .Testing 

The  amplitude  statistical  routines  perform  two  tests.  First,  a  test  of 
independence  between  the  noise  and  maximum  amplitude  distribution  is 
carried  out,  then  the  maximum  amplitudes  of  each  antenna  are  tested.  The 
Most  Probable  Amplitude  (MPA)  and  the  maximum  amplitude  (Amax)  of 
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each  spectrum,  averaged  over  a  group  of  drift  data,  are  used  in  these  tests. 
The  Most  Probable  Amplitude  (MPA)  and  Amax  are  averaged  over  a  drift 
data  group  and  their  mean  and  standard  deviation  are  obtained. 

A  drift  data  group  is  chosen  in  the  ODDAMENU.ONL  file  at  option  *042.  The 
value  entered  specifies  a  time  interval  in  minutes  for  the  gathering  of  drift 
data  before  the  statistical  testing  is  done.  For  example,  dialing  in  a  5 
would  fo^ce  DDA  or  ODDA  to  carry  out  a  statistical  quality  control  test  after 
collecting  N  amount  of  subcases  contained  within  a  5  minute  period. 

3.1.1  Independence  Test 

In  the  amplitude  graph,  the  last  two  columns  (not  labeled)  are  active  and 
display  the  failure  rate  for  an  independence  test  performed  between  the 
noise  and  signal  distributions.  Consider  the  problem  of  testing  whether  a 
signal  amplitude  can  be  considered  as  being  related  to  the  noise  level  of 
the  spectrum.  Hence,  we  would  like  to  test  the  hypothesis  that: 

Ho:  pi  =  P2 

(1) 

Hi:  pi  *  P2 
where 

Pi  =  Mean  MPA  averaged  over  a  group  of  subcase  and 
four  antennas. 

oi  =  Standard  deviation  of  MPA  averaged  over  a  group  of  height 
spectrum  and  four  antennas. 

P2  =  Mean  maximum  amplitude,  Amax,  averaged  over  a  group  of 
subcase  and  four  antennas. 

02  =  Standard  deviation  of  the  maximum  amplitude  averaged 

over  four  subcases  and  four  antennas. 

We  make  the  assumption  that  the  sampling  distribution  of  MPA  and  Amax 
are  approximately  normally  distributed  with  mean  pi,  P2  and  standard 
deviations  oi,  02,  respectively. 
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Statistically  the  mean  of  the  signal  (i.e.  maximum  amplitude  of  the 
spectrum)  distribution  can  approach  the  mean  of  the  MPA  and  still  be 
considered  a  separate  distribution  as  long  as  P2  does  not  belong  to  the 
distribution  of  pi,  i.e.  p2  cannot  be  less  than  or  equal  to  the  Za/2  confidence 
level  of  p i,  in  such  a  case  pi  and  P2  are  said  to  be  independent.  The  critical 
region  for  each  distribution  may  be  given  as 


Zo/2 


x-^o 

o/Vn 


where 

a  =  Level  of  significance 

n  =  No.  of  samples 

x  =  Testing  mean  =  P2 

po  =  Mean  of  test  distribution 

o  =  Standard  deviation  of  test  distribution 


(2) 


Since  the  amplitudes  of  sources  must  always  be  greater  than  the  noise 
then  the  only  criterion  which  needs  to  be  satisfied  is  that 


*12>'11+Za«^  O) 

As  long  as  this  criterion  is  satisfied  the  two  distributions  may  be 
considered  to  be  independent.  If  this  condition  is  not  met  a  failure  count  is 
recorded.  Since  the  overall  distribution  for  MPA,  considered  here  as  the 
noise  level,  and  the  maximum  amplitude  are  being  compared,  this  test 
allows  us  to  determine  how  close  the  source  signals  are  to  the  noise.  As 
such,  the  test  for  independence  can  be  used  as  a  data  quality  check.  The 
method  here  is  only  used  for  amplitude  maximum  comparisons,  in  effect 
this  analysis  could  be  done  on  each  spectral  line  as  an  indication  if  source 
signal  exists  above  the  noise  level.  Currently  a  fixed  noise  level  is  defined 
and  any  source  signal  below  this  value  would  be  ignored.  This  new 
method  allows  this  threshold  to  be  lowered  (or  raised)  statistically  and  still 
maintain  a  high  level  of  data  integrity.  It  is  not  incorporated  into  the 
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onsite  ODDA  quality  control  since  testing  every  spectral  line  would  take 
time,  but  as  for  playback  analysis  from  tape  (i.e.  using  DDA),  such  a  system 
could  prove  beneficial  in  detecting  "weaker"  signals. 

Currently,  the  independence  test  is  only  displayed  and  recorded  as  an 
indicator  of  data  integrity.  DDA  or  ODDA  will  not  use  this  result  to 
automatically  delete  data  sets.  The  operator  must  decide  on  the  value  of 
this  failure  rate,  whether  to  keep  or  discard  data.  If  the  independence  test 
failure  is  greater  than  50%  in  AVE,  then  even  if  a  large  number  of  sources 
are  detected,  this  data  set  would  be  considered  to  be  bad  and  not  much 
confidence  can  be  placed  on  the  resulting  skymap  and  velocities. 

The  second  test  performed  is  similarly  a  two-sided  test,  only  this  time  the 
maximum  amplitudes  of  each  antenna  are  tested.  The  antenna  with  the 
highest  averaged  Amax  is  used  as  a  reference  antenna  to  test  all  other 
antennas.  However,  the  averaged  mean  standard  deviation  o 2  is  used 
rather  than  the  standard  deviation  calculated  for  the  reference  antenna. 
The  test  distribution  then  constructed  should  allow  all  the  amplitudes  to 
fall  within  this  region.  Any  pj  which  does  not  fall  within  the  accepted 
region 


°2  a2 

^rcf  'Za/2^<  <  ^ref  +  Za/2  ^  (4) 

is  regarded  as  a  failure. 

Actually,  since  pref  is  the  mean  maximum  amplitude  possible  for  all 
antennas  then  only  the  lower  part  of  this  distribution  need  be  tested. 
Hence,  the  only  criterion  that  need  be  satisfied  is: 

a2 

Pj  >  Pref  -  .  (5) 

If  satisfied  pj  belongs  to  the  distribution  and  is  consistently  between 

Pref  -3a  <  Pj  <  Prof  +  3a  .  (6) 
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3.2 


Four  methods  were  devised  to  determine  if  phase  errors  exist  in  each 
antenna.  The  first  method  used  the  phase  differences  between  antennas  to 
determine  the  integrity  of  the  phase  measurement.  Considering  the 
geometry  of  the  receiving  antenna  array,  if  $j  is  the  phase  recorded  for  a 
spectral  line  for  antenna  i  then  the  equations  that  need  to  be  satisfied  may 
be  written  down  as: 

$2  +  4>3  +  -  3<|>i  =0  1 

<j)5  +  <j)6  +  <>7  -  3<J>1  =  0  2 

2<j>5  +  $6  -  3<J)2  =03  (7) 

2<j>6  +  <(>7  -  3<J>3  =0  4 

2<J>7  +  05  -  3<|>4  =0  5 

Since  we  have  seven  unknowns  and  only  five  equations,  we  cannot  solve 
for  the  individual  <j>'s.  But  we  can  test  the  distribution  for  each  equation 
from  data  gathered  and  by  recording  what  combination  of  equations  fail 
we  may  determine  what  possible  combination  of  antennas  may  not  be 
functioning.  Table  1  shows  the  equation  failure  combination  and  which 
antenna  combination  can  cause  these  equations  to  fail.  A  failure  is 
recorded  as  1,  if  on  the  other  hand  the  equation  satisfies  the  criteria  then  0 
is  displayed,  e.g.  if  equation  1  and  equation  2  do  not  fall  within  the 
distribution  required,  but  all  other  equations  are  satisfied,  then  Antenna  1 
is  in  error,  etc.  Initially  Phases  are  compared  only  for  -3  to  3  Doppler 
lines. 

Identifying  the  binary  code  for  the  testing  of  phases  from  antennas  would 
then  give  an  indication  as  to  which  antenna  or  antenna  combination  may 
be  giving  a  problem.  The  method  does  have  its  drawbacks.  These  are: 

1.  Seven  antennas  would  need  to  be  sampled  on  a  regular  basis  in 
order  for  this  method  to  work  for  a  quality  control  check,  which 
should  be  done  every  time  a  measurement  is  taken. 

2.  If  only  one  antenna  fails  then  identification  can  be  made.  But 
for  certain  combinations  of  two  or  more  antennas  failing  a  large 
number  of  possible  combinations  exist. 

3.  As  shown  by  large  scale  statistics  produced  by  C.  Dozois,  the 
distributions  of  the  equations  are  not  centered  on  0  and  also 
tend  to  shift  in  time. 
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Table  1.  Statistical  Binary  Error  Code  and  Associated  Bad  Antenna(s). 


Bad  Antennas 

Bad  Equations 
1-2-3-4-5 

Binary  Code 

Antenna  -  1 

—A 

1 

1 

—A 

1 

t 

o 

1 

1 

o 

1 

o 

3 

"  -2 

1 

1 

o 

i 

• 

—A 

1 

1 

o 

1 

1 

o 

5 

•  -  3 

1 

• 

o 

• 

1 

o 

1 

1 

—A 

1 

1 

o 

9 

"  -4 

mmA 

1 

1 

o 

1 

• 

o 

1 

1 

o 

1 

1 

-A 

17 

•  -5 

o 

1 

1 

—A 

1 

1 

• 

1 

o 

1 

1 

22 

"  -  6 

14 

"  -7 

0--1--0--1 --1 

26 

•  -  1&2 

i 

i 

» «a 

i 

• 

i 

» 

o 

1 

1 

o 

7 

"  -  1&3 

1 

• 

• 

• 

o 

• 

1 

-A 

1 

o 

1 1 

"  -  1&4 

1-1- -0-0-1 

19 

-  -  2&3 

1-0--1--1-0 

13 

-  -  2&4 

•jk 

i 

% 

O 

1 

1 

1 

1 

o 

1 

1 

— V 

21 

-  -  3&4 

f 

1 

• 

1 

o 

1 

1 

o 

1 

1 

25 

-  -  2&3&4 

1  —0—1  —1—1 

29 

-  -  1&6.3&6, 

2&6.2&3&6.1&2&3, 

1  &2&6.1  &2&3&6  1-1-1-1-0  15 

-  -  1&5.2&5, 

4&5.1&2&4.1&2&5  1-1-1-0-1  23 

"  -  6&7,5&7, 

5&6.1  &2&7,5&6&7  0--1--1-1--1  30 

"  -3&7.1&7, 

3&4&7,2&3&4&5  1-1-0-1-1  27 

All  other. 

combinations  1—1—1— 1—1  31 
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It  soon  became  apparent  that  testing  of  antennas  for  a  quality  control 
method  required: 

(a)  Testing  be  continuous;  for  each  measurement  antennas  should  be 
tested. 

(b)  Antennas  should  be  tested  separately  and  not  depend  on  an 
antenna  combination  outcome. 

Again  referring  to  previous  statistical  data,  it  became  apparent  that  the 
phase  on  each  antenna  should  be  distributed  randomly  over  a  large 

number  of  samples.  Similarly  producing  and  o  parameters  for  the  phases 
of  all  antennas  for  a  number  of  cases,  it  was  considered  reasonable  that  the 
distribution  of  these  phases  are  random,  since  they  tend  to  produce  a 

boxcar  spectrum.  Hence,  the  ability  to  test  for  randomness  in  the  phase 
data  may  be  a  good  test  for  the  quality  of  phase  data  being  observed. 
Thus,  a  test  for  randomness  was  introduced  into  the  program. 

Two  tests  for  randomness  are  currently  used.  Both  methods  may  be  found 
in  K.V.  Bory  (1975)1. 

3.2.1  Run  Test 

One  of  the  simplest  methods  used  to  test  the  randomness  of  a  set  of  data  is 
to  note  the  signs  of  differences  in  the  magnitudes  of  consecutive 
measurements.  As  an  example  given  a  data  series  represented  by  <j>i  for 
phase  then: 

sign  (4>i+i  -  4>i)  =  +1  if  <j>i+l  >  0i  (8) 

sign  (0i+i  -  $0  =  -1  if  fo+i  <  <|>i 

if  =  <}>i  the  zero  cases  are  ignored. 

Then  by  observing  the  sequence  of  consecutive  plus  signs  and  minus  signs 
(which  is  termed  a  run),  it  is  possible  to  use  this  number  as  a  test  statistic, 
e.g.  of  a  set  of  runs. 

♦  90  162  308  308  102  110  125  205  200  10  60  15  5  100  200  30 

sign  -1  -1  0  +1  -1  -1  -1  +1  +1  +1  -1  +1  +1  -1  -1  -1 

N2i  Nli  N22  Nl2  N23  Nl3  N24 


1  Karl  V.  Bory,  Statistical  Models  in  Applied  Science,  John  Wiley  and  Sons,  New 
York,  New  York,  et.  pp.  225-238,  1975 
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The  row  above  has  seven  runs. 


N1  =  the  total  number  of  plus  signs  =  3 

N2  =  the  total  number  of  minus  signs  =  4 

U  =  the  total  number  of  runs  =  7 

then  the  expected  value  and  variance  of  U  can  be  given  as: 


E(U)  =  1  + 


2N1N2 

N1+N2 


(9) 


Var(U)  = 


2N1N2(2N1N2-N1-N2) 
(N 1  +N2)2(N  1 +N2- 1 ) 


This  test  can  be  conducted  to  determine  a  value  U  such  that 


(10) 


Pr(U  £  u)  *  0.997. 

If  this  condition  is  satisfied  then  data  is  random  otherwise  something  is 
wrong  with  the  phase  of  the  antenna. 

The  second  method  to  test  for  randomness  is  termed  the  Label  Test. 


3.2.2  Label _ Tsst 

Consider  now  that  the  observations  Xj  are  given  labels  Y  =  i.  Clearly  if  the 
sequence  of  observations  on  X  constitutes  a  random  sample,  then  the 
measured  variable  X  and  label  variable  Y  are  statistically  independent.  For 
example,  consider  the  data 

Number  of  X  100  90  3  5  62  7  3  8  8  102  306  250 

Orders  Y1  23456789 

Now  since  X  and  Y  appear  in  pairs  the  test  for  independence  by  use  of  the 
sample  correlation  coefficient  can  be  applied  to  X  and  Y  to  check  the 
randomness  of  the  observations  on  X. 

The  sample  correlation  coefficient  may  be  written  as 


n 


(H) 


X.Y.-XY 
r  = 

i= 

where 

Sx  = 

Sy  = 

Y  = 

X  = 

The  correlation  coefficient  may  be  tested  using  the  t  variable  as 

t  =  r[(n-2)/(l-rJ)]1/2  (12) 

Since  over  four  subcases  n  =  degrees  of  freedom 

=  n  -  2  =  256  -  2  =  254  >  29 

then  it  is  possible  to  use  the  infinity  option  of  the  t-distribution.  Hence,  at 
1%  critical  t  value  ti/2  =  2.576  for  infinity.  Then  the  calculated  t  value 
obtained  from  equation  (1)  must  fall  within  the  region 

-2.576  <  t  <  2.576  (13) 

for  the  test  to  be  accepted  and  data  to  be  random.  DDA  and  ODDA  quality 
control  actually  tests  Itl  <  2.576.  Since  the  observations  are  compared  to  a 
monitonically  increasing  variable  Y,  the  Label  test  is  sensitive  to  consistent 
changes  in  the  phase.  Consider  the  case  of  a  system  adding  A$  to  each 
phase  measurement,  the  total  phase  recorded  would  be  =  0  +  A«|>.  If  <j>  is 
random  the  run  test  may  not  detect  this.  The  Label  test,  however,  would 
observe  the  characteristics  of  A<f>  component  since  the  correlation  between 
variables  X  and  Y  would  increase. 


1  x  y 

variance  of  x 

variance  of  y 
mean  of  Y 
mean  of  X 
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The  fourth  method  to  analyze  the  integrity  of  the  phase  being  recorded 
attempts  to  quantitatively  determine  the  phase  error  consistently 
observed  on  each  antenna/cable/switch  system.  The  method  uses  the 
azimuth  and  zenith  values  calculated  for  each  reflection  source  point  to 
determine  the  phases  on  each  antenna.  These  calculated  phases  are 
compared  to  the  phases  used  in  the  least  squares  fit  and  the  differences 
A<j>i  are  recorded.  After  100  sources  are  processed  in  this  way,  the  phase 
differences  are  averaged  and  the  averaged  phase  difference  A<J>avei  is 
introduced  back  into  the  next  set  of  phases  obtained  from  the  drift  data. 

This  method  has  3  operational  modes  that  may  be  selected  using  option 
074  in  ODDAMENU.ONL. 

Mode  1:  Phase  correction  on,  but  do  not  compensate  for  errors. 

This  mode  is  set  active  by  selecting  4  or  5  in  option  074.  Without 
compensating  for  phase  differences  observed  this  mode  is  useful  in 
collecting  statistics  of  phase  errors  observed  in  each  antenna/cable/switch 
system.  The  average  phase  differences  collected  over  100  sources  are 
averaged  with  the  previous  value.  The  statistic  is  therefore  accumulative 
and  can  be  represented  as: 

^Ei  =  (<t>Ei  +  A<j>avei)/2  (14) 

where 

<j>Ei  =  phase  error  in  antenna/cable/switch  system  i 
A<t>avei  =  phase  differences  in  antenna/cable/switch  system  i, 
averaged  over  100  sources 

Processing  drift  data  with  this  mode  set  active  will  produce  a  table  with 
phase  errors  for  each  antenna/cable/switch  system  given  as  a  function  of 
frequency.  The  table  is  housed  in  the  QUCNTRL.DAT  file. 

The  phase  error  displayed  in  the  quality  control  represents  the 
accumulative  error.  In  this  way,  the  operator  may  quickly  identify  if  the 
phase  errors  are  acceptable  for  the  drift  calculation.  If  errors  of  20°  or 
more  are  observed  consistently  in  the  data  set,  phase  correction  with  error 
compensation  may  be  needed  (i.e.  use  modes  2  or  3). 
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Mode  2:  Phase  correction  on,  compensate  for  errors. 

This  mode  is  set  active  by  selecting  2  or  3  in  option  074.  Unlike  mode  1, 
this  option  will  not  give  a  true  representation  of  the  phase  errors  on  each 
antenna/cable/switch  system  since  the  original  drift  phases  are 
continuously  being  updated.  The  method  tries  to  minimize  the  error 
recorded  in  <|>avei  by  adjusting  the  original  phases  used  in  the  least  squares 
fit.  The  adjusting  or  correcting  phase  offset  is  given  as  <|>ci  and  is  calculated 
for  certain  criteria.  The  criteria  are: 

for 


l^aveil  >  l(<j>Ci)/2l 


(15) 


and 


l^aveil  >  0-04  radians  (16) 

then  the  calculation  of  <X>ci  is  stated  as: 

=  4»Ci  +  ($avei)/2  (17) 

Criterion  of  equation  15  sets  up  an  error  limit  for  the  100  source  average, 
otherwise  <{>ci  could  increase  to  values  that  overestimate  the  actual  error  on 
the  system  i. 

Criterion  of  equation  16  minimizes  the  error  adjustment  of  <j>ci  for  each  i 
system.  If  the  4>avei  value  is  less  than  approximately  5°,  no  update  of  the 
correcting  phase  offset  is  done. 

Since  the  phase  correction  is  done  as  data  are  being  gathered,  the  first  few 
hundred  sources  would  not  change  much  since  the  statistical  data  are  still 
being  collected. 

In  this  mode  the  quality  control  display  shows  $avei-  If  the  phase 
correction  is  working  well,  the  value  of  <f>avei  averaged  over  100  height 
spectra  should  show  small  phase  errors,  at  least  less  than  10°. 
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Mode  3:  Phase  correction  on,  calibration  mode. 

This  mode  is  set  active  by  selecting  6  in  option  074.  This  mode  is  intended 
for  use  with  phase  errors  calculated  from  calibration  data  or  large  scale 
statistics  using  mode  1.  The  phase  errors  on  each  i  system  as  a  function  of 
frequencies  are  supplied  in  the  QUCNTRL.DAT  file.  DDA  will  use  only  these 
errors  to  adjust  phases  before  the  least  squares  fit  is  done.  As  in  mode  2, 
the  <J>avei  are  displayed  to  ensure  that  the  phase  errors  used  do  minimize  the 
phase  differences  calculated  from  the  drift  data. 

Analysis  of  Phase  Correction  Method  Using  Simulated  Data 

To  test  the  DDA  phase  correction  method,  drift  data  were  simulated  for 
known  source  positions  and  plasma  velocities.  An  example  of  the  simulated 
drift  data  is  given  in  Figure  2.  The  drift  data  were  calculated  for  the  seven 
antenna  setup  at  Millstone  Hill,  Westford,  MA,  with  O-ray  echoes  recorded. 
The  input  Digisonde  parameters  corresponding  to  this  configuration  would 
be: 

X  =  0  =  no  phase  code 

L  =  B  =  8  antenna  bins 

Z  =  2  =  7  antenna  setup  and  sum  of  antennas 

N  =  5  =64  spectral  lines 

R  =  2  =  100  Hz  repetition  frequency 

W  =  5  =  132psec  pulse  width 

Each  reflection  surface  was  given  plasma  velocities  of: 

Vz  =  lOm/s  =  vertical  velocity 

Vh  =  lOOm/s  =  horizontal  velocity  magnitude 

Az  =  90  Deg  =  azimuthal  angle  of  horizontal  velocity 

Random  noise  was  also  introduced  into  each  spectrum  to  make  the  drift  data 
more  realistic  and  allow  DDA  to  calculate  a  good  MPA  value.  The  position  of 
the  sources  are  shown  in  the  sky  map  given  in  Figure  3.  Groups  of  sources 
located  at  30°  increments  in  azimuth  and  at  10  and  20°  in  zenith  are  shown. 

A  phase  error  was  introduced  into  the  antenna/cable/switch  system  for 
antenna  #2.  The  drift  data  generated  was  then  analyzed  with  DDA.  Figure  4 
shows  the  DDA  calculated  phase  errors  plotted  against  the  known  phase 
error  in  antenna  #2.  Figure  4a  displays  the  phase  difference  averaged  over 
the  last  100  sources,  4>avei»  while  Figure  4b  displays  the  accumulated  phase 
error  statistic  <J>Ei  calculated  over  «600  sources. 
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STATION 
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Figure  2.  Simulated  Drift  Data,  Raw  Spectral  Output. 
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Figure  3.  Skymap  of  Simulated  Data  Points. 
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Cal.  Adjusted  Phase  per  100  sources 
Simulated  Data 

Vz=  lOm/s  Vh  =  100m/s  Az=90Deg. 


-50-f-n 

-50 


-3D  -10  10  30 

Actual  Phase  Error  on  Ant.  2 


O  Antenna. 2 
A  Antenna.3 
OAntenna.4 
+Antenna.5 
xAntenna.0 
^•Antenna.7 


Cal.  Total  Adjusted  Phase 
Simulated  Data  (Approx.  600  sources) 
Vz=  lOm/s  Vh=  lOOm/s  Az=90Deg. 


A 

cu-io- 


O  Antenna.2 
a  Antenna.3 
OAntenna.4 
+Antenna.5 
X  Antenna. 0 
Antenna. 7 


-30  -10  10  30 

Actual  Phase  Error  on  Ant.  2 


Figure  4.  Calculated  Phase  Errors  Using  Mode  1. 


The  dashed  line  in  both  figures  indicates  those  data  points  where  the 
calculated  and  actual  phase  errors  should  agree.  It  is  important  to  note, 
for  example,  that  a  known  phase  error  of  -10°  DDA  calculates  a  phase  error 
Oei  of  approximately  +10°.  The  change  in  sign  results  since  DDA  calculates 
the  compensation  error  to  be  added  to  the  phase  of  antenna  #2.  To  obtain 
these  results,  mode  1  was  used  so  that  the  phases  used  in  this  least 
squares  fit  were  not  adjusted. 

The  results  in  Figure  4  show  that  the  phase  correction  scheme  is  able  to 
calculate  the  error  on  antenna  #2.  The  corrected  phase  error  d>Ei  is 
underestimated  for  larger  phase  errors  introduced  in  antenna  #2.  At  the 
same  time,  the  other  antenna  systems  display  increasing  phase  errors  as 

well.  The  reproduction  of  the  phase  errors  determined  over  600  sources 
(Figure  4b)  and  those  determined  from  only  100  sources  (Figure  4a) 
indicates  that  averaging  over  100  sources  is  a  good  estimate  of  the  phase 
errors  observed  on  each  antenna. 

Figure  5  displays  the  phase  correction  scheme  when  the  DDA  phase 
correction  is  active  and  also  updating  the  drift  phases  before  the  least 

squares  fit.  The  results  in  this  figure  were  obtained  using  Mode  2. 

Comparing  Figure  5b  and  Figure  4b  it  is  clear  that  the  determined 
corrected  phase  errors  <J>Ei  and  <Dci  are  not  the  same.  This  is  to  be  expected 
since  in  Mode  2  the  d>ci  are  being  used  to  compensate  for  errors  in  the 
phase  before  a  least  squares  fit  is  done.  Hence,  the  accumulation  of 

statistics  is  affected  by  the  updating  of  the  input  phase. 

The  phase  difference  <>avei  of  the  last  100  sources  shown  in  Figure  5a 
displays  small  phase  errors  for  each  antenna  system.  This  shows  that  the 
DDA  phase  correction  scheme  is  very  efficient  in  compensating  for  phase 
errors  in  antenna/cable/switch  systems.  The  results  shown  here  give 
confidence  in  the  technique  for  simulated  data.  However,  when 
considering  using  drift  data  collected  from  radio  waves  reflected  from  the 
ionosphere  the  scheme  may  not  be  able  to  give  a  true  representation  of 
phase  errors  on  each  antenna  system,  however,  it  can  compensate  for  large 
phase  errors  and  keep  these  errors  small. 

Further  work  in  using  these  schemes  of  the  quality  control  on  real  data  is 
continuing.  The  results  of  how  good  the  current  quality  control  system  is 
depends  on  the  variety  and  accumulation  of  ionospheric  data  and  statistics. 
This  is  an  ongoing  analysis  effort  to  assess  and  improve  the  ODDA/DDA 
quality  control  package.  The  following  section  describes  some  preliminary 
experimental  analysis  methods. 
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Cal.  Phase  Error  Cal.  Phase  Error 


Cal.  Adjusted  Phase  offset  per  100  sources 
Simulated  Do*a  (Approx.  600  sources) 
Vz=10m/s  Vh  =  100m/s  Az  =  90Deg. 

(Phase  Correction  Active) 


O  Antenna. 2 
a  Antenna. 3 
O  Antenna.4 
■+■  Antenna.5 
X  Antenna. 6 
>KAntenna.7 


Cal.  Total  Adjusted  Phase 
Simulated  Data  (Approx.  600  sources) 
Vz=  lOm/s  Vh=100m/s  Az=90Deg. 
(Phase  Correction  Active) 


Figure  5.  Calculated  Phase  Error  Corrections  Using  Mode  2. 
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4 .  EXPERIMENTAL  ANALYSIS 


The  DDA  quality  checking  is  consistently  being  tested  when  data  is 
sampled.  In  most  cases  ,data  types  where  antennas  are  known  to  be  bad 
are  analyzed.  A  few  examples  are  presented  here  to  illustrate  the 
effectiveness  of  the  Quality  Control  System. 

'  Figure  6  displays  amplitude  statistics  for  data  collected  at  Qaanaaq  in 
1988.  In  this  drift  mode  only  four  antennas  were  used.  It  is  apparent  that 
from  day  88177  to  88180  the  amplitude  of  antenna  6  became  weak.  A 
data  sample  of  *>3100  drift  cases  around  this  time  was  analyzed  by  ODD  A. 
The  amplitude  and  phase  testing  for  this  quality  control  are  shown  in 

Figure  7. 

Figure  7  displays  the  %  failure  counts  obtained  over  groups  of  100  cases  of 
drift.  Recall  that  each  time  the  maximum  amplitude  of  the  spectral  lines 
averaged  over  four  subcases,  for  each  antenna,  does  not  satisfy  certain 

criteria,  a  failure  is  recorded.  The  number  of  failures  recorded  then 
constitutes  a  failure  count.  After  100  cases  are  sampled,  this  failure  count 
is  averaged  with  the  previous  100  case  average,  and  the  result  is  the  % 
failure  count.  For  example,  a  large  %  indicates  that  the  antenna  is  not 
functioning  properly.  On  average  all  antennas  should  display  failures  for 
amplitude  and  phase  of  less  than  20%.  This  figure  was  arrived  at 
statistically  by  sampling  a  large  data  set.  However,  it  is  still  recommended 
that  a  Quality  Control  analysis  is  done  on  the  data  for  each  station  where 
DDA  is  to  be  installed,  since  this  threshold  of  20%  could  vary  from  station 
to  station.  During  normal  ionosonde  operations  if  the  %  failure  counts  are 
higher  than  this  threshold,  then  the  antennas  should  be  checked. 

Recall  that  the  drift  mode  being  used  for  this  analysis  only  used  the  four 

outer  antennas  (i.e.  ANT.  1,  5,  6,  7).  Figure  7a  displays  results  of  the 

amplitude  test.  It  is  clear  that  around  2000  cases  antenna  6  began  to  fail. 
Within  100  cases  the  failure  rate  had  increased  to  40%,  while  the  testing  of 
the  other  antennas  show  failure  counts  consistently  below  10%.  In  Figure 
7b  a  slight  increase  in  the  failure  count  for  the  phase  testing  method  is 
observed,  although  the  failure  count  remains  below  15%.  In  addition,  after 
2000  cases  it  is  not  surprising  that  antenna  6  consistently  has  the  lowest 
failure  rate  for  phase  testing,  since  this  antenna  is  not  gathering  data,  then 
we  expect  more  noise  to  be  introduced  in  the  spectra,  and  so  the  test  for 
randomness  would  be  more  easily  satisfied. 
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Figure  6. 
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Testing,  Qaanaaq  Days  88177  through  88180. 
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Figure  8  displays  the  number  of  occurrences  (converted  to  percent)  of  each 
of  the  phase  0  to  31  (i.e.  the  five  MSB  are  used).  During  days  87239- 
87324  a  problem  was  detected  in  the  phases  since  the  values  are  not 
evenly  distributed  and  certain  phases  dominate.  Figure  9  shows  the  DDA 
quality  control  testing  for  days  87291  through  87293.  From  Figure  9b,  the 
failure  counts  for  phase  testing  are  high,  with  an  average  value  of 
approximately  26%.  In  comparison,  the  amplitude  testing  (Figure  9a) 
shows  no  problems  with  the  system. 

Figure  10  displays  the  DDA  quality  control  testing  on  days  87328  through 
87330,  after  the  problem  with  the  phase  was  fixed.  Both  testing  methods 
return  values  below  10%.  The  difference  in  failure  counts  between  Figure 
9b  and  10b  is  apparent. 

As  with  all  statistics,  one  always  prefers  the  largest  possible  data  sample 
to  test  statistical  methods.  While  the  data  analyzed  for  the  verification  of 
the  DDA  quality  control  method  were  large,  we  still  consider  this  analysis 
as  preliminary.  However,  it  is  clearly  indicated  from  these  results  that  the 
methods  do  allow  us  to  determine  if  problems  in  the  phases  and 
amplitudes  of  the  drift  spectrum  exist.  What's  more,  only  100  cases  of 
drift  are  required  to  determine  if  a  problem  exists.  Again,  how  robust 
these  techniques  are  under  changing  ionospheric  and  instrumentation 
conditions  is  still  in  question  and  is  continuing  to  be  investigated. 
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Figure  8.  Number  of  Occurrences  of  the  5  MSB  of  Phase, 
Qaanaaq  Digisonde  256 
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Figure  9. 
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Figure  10.  Number  of  Failure  Counts  for  Phase  Testing  in  Quality 
Control,  Qaanaaq  Digisonde  256,  Days  87328  through  87330. 


27 


